High definition virtual reality (VR) and augmented reality (AR) are becoming pervasive. Head-mounted display devices such as virtual reality goggles (VR goggles) and AR glasses can provide a highly immersive experience. Modern cameras and rendering engines such as NVIDIA GeForce GPUs produce high resolution photorealistic imagery. In some such systems, video is generated and encoded by a camera, GPU or other source. The source can often produce video in interactive response to the user's pose and/or other user input (e.g., direction of gaze). The video is transmitted to a local processor associated with VR goggles or AR glasses worn by a user. The VR goggles or AR glasses receive and display the transmitted video frames at regular intervals.
In more detail, in order to create the perception of movement, many display devices such as VR goggles operate by having the screen redrawn at a regular periodic rate known as the display's “refresh rate”. The display device redraws the image tens or hundreds of times each second as determined by the refresh rate. The refresh rate may be fixed or configurable for a particular display device depending on its type and capabilities, and typically is in the range of 60 to 200 frames per second (fps). For example, when the VR goggles or AR glasses operate at a 60 frames per second (fps) refresh rate, they display a new image each 16.67 milliseconds. This time period is sometimes called a “refresh interval” because the display refreshes with a new image frame at the end of each such time interval. The graphics rendering engine or other source is responsible for generating and sending image frames at the requisite frame rate so the display will have the next image frame available when it needs to display it.
It takes a certain finite amount of time to render and display each new image. Such delay is often referred to as “latency”. Latency can especially important in VR and AR applications where the presented images need to keep up with changes in the user's gaze, head direction and the like. For example, if the user quickly turns her head to the left, the display must be able to track the quick head turning and display a series of images appropriate to the user's changing head orientations. The user will instantly notice the delay if the display falls behind. Such delay can lead to lack of realism, disorientation and even nausea.
To reduce latency, the rendering engine typically operates at least one image frame ahead and stores the next image frame in a display buffer memory local to the display so the frame is present when the display needs it. Some display systems use more than one display buffer. For example, in some computers, two display buffers are used so that while the contents of a first display buffer are being written to the display device, the computer or GPU can write a newly generated image to the second buffer. The display system can be configured to switch between the buffers in an alternating manner.
Fast rendering hardware is used to keep up with the display refresh rate, and various techniques are known for reducing image complexity and/or resolution to ensure image rendering does not fall behind. Once a next frame is stored in the display buffer memory, display refresh hardware typically reads the contents of the display buffer memory and provides it to the display device at the requisite refresh time intervals demanded by the display.
Real-time graphics artifacts such as those described above are more frequently observed in environments where rendering engine must communicate with the display via a remote communications link such as the Internet and/or a wireless connection (e.g., WIFI, GSM cellular telephone data network, etc.). Since signals do not travel instantaneously through space, such links will inherently exhibit some delay or latency. Additionally, well known video compression techniques can be used to ensure that there is enough time to transmit images of desired resolution over the bandwidth the communications link offers.
Due to variable transmission characteristics of such communications links, latency may vary from one transmission to the next. Such variability is typically referred to as jitter, meaning deviation from true periodicity of a presumably periodic signal (i.e., transmitted at the periodic display refresh rate). Such jitter can cause some images or parts of images to arrive quickly, whereas other images or parts of images may take a longer time to traverse the link from the source to the destination. In these environments, the amount of network jitter may increase when the path from the rendering engine to the VR goggles or other display includes many hops and/or many alternate paths, when transmission errors and/or bandwidth limitations require retransmission and/or other transient network degrading conditions occur.
Screen tearing and/or other real-time graphics artifacts can occur when the process by an image renderer of writing an image into the display buffer is not adequately synchronized with the operation of the display device. “Screen tearing” results when parts of two different images are simultaneously displayed on the display device. Screen tearing can occur when the content of a display buffer from which the display is drawn is changed before the display device draws a complete frame. For example, if the rendering engine begins writing a new image to a buffer while the display device is still reading an earlier image out of the same buffer, one part of the display will show a part of the earlier image while another part of the display show a part of the new image. Another real-time graphics artifact known as “stutter” occurs when writing into the display buffer is delayed with respect to reading according to the refresh rate of the display, and the display reads and redraws stale (already previously displayed) content from the buffer.
As cloud-based graphics rendering gains in popularity and therefore as more users begin to view remotely rendered video images on a local display device, it becomes even more important to address issues such as the appearance of real-time graphics artifacts noted above in the displayed video. Cloud-based rendering is only one example application; numerous other applications exist where the rendering of the video occurs at a first location and where the display of that rendered video at a second, different location should be performed without generating real-time graphic artifacts such as those described above.
Real-time graphics artifacts due to network jitter is not an issue that is restricted to certain type of processors, and may occur when the rendering engine and/or the displaying client device have CPU, GPU and/or any other type of processor capable of image generation and display. In the past, GPU functions were often limited to a well-defined set of graphics operations. This changed when GPUs became programmable. For example, in 2001, NVidia released its GEForce3 NV20 GPU providing programmable vertex and pixel shaders. Later, Nvidia developed the CUDA® parallel computing platform and programming model for general computing on graphical processing units (GPUs). With CUDA®, developers have been able to dramatically speed up computing applications by using GPU processing capabilities to perform computing tasks in addition to graphics tasks. Thus, GPUs have become a highly versatile and powerful processor handling the processing for a bulk of modern video games and complex applications, particularly those that involve substantial video and/or image rendering. The GPUs (and perhaps also other types of processors) have evolved to be capable of real-time rendering of highly complex detailed images resulting from highly complex calculations. However, in order to improve the user experience associated with cloud-based image rendering for use by user's not directly connected to the rendering engines, it is desirable that real-time graphics rendering artifacts such as those described above be eliminated or at least reduced.
Thus, there is a need for systems and methods for improving the display of images on a local display device when the images are initially rendered in a remote device.